For continuing tasks, average cost Markov decision processes have welldocumented value and can be solved using efficient algorithms. However, it explicitly assumes that the agent is risk-neutral. In this work, we extend risk-neutral algorithms to accommodate the more general class of dynamic risk measures. Specifically, we propose a relative value iteration (RVI) algorithm for planning and design two model-free Q-learning algorithms, namely a generic algorithm based on the multi-level Monte Carlo (MLMC) method, and an off-policy algorithm dedicated to utility-based shortfall risk measures. Both the RVI and MLMC-based Qlearning algorithms are proven to converge to optimality. Numerical experiments validate our analysis, confirm empirically the convergence of the off-policy algorithm, and demonstrate that our approach enables the identification of policies that are finely tuned to the intricate risk-awareness of the agent that they serve.

Neural Information Processing Systems http://nips.cc/

This work was done while M. Menart was at The Ohio State University.

The code is availableat tt s t s q.

Next, we would like to prove the opposite direction. All LeakyReLU activations have a slope of 0.02 for negative values except when we use a classic discriminator for single image training, for which we use a slope of 0.2. Additionally, the generator's last conv-block activation at each scale is Tanh instead of ReLU and the discriminator's last We clip the gradient s.t it has a maximal L2 norm of 1 for both the generators and Batch sizes of 16 were used for all experiments involving a dataset of images. At test time, the GPU memory usage is significantly reduced and requires 5GB. In this section, we consider training our method with a "frozen" pretrained ResNet34 i.e., optimizing If the problem could be learned with a "small enough" depth, our method would benefit from even As can be seen, our method yields realistic results with any batch size.

However, its application to Q-learning has been limited due to the presence of the max-operator, which makes the associated ODE model a complex nonlinear system. In contrast, the associated ODE of TD learning for policy evaluation is a linear system, whose asymptotic stability is much easier to analyze in general.